Aligning Pre-Operative Scan Images To Real-Time Operative Images For A Mediated-Reality View Of A Surgical Site

ABSTRACT

A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon&#39;s head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon&#39;s head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/749,963 filed on Jan. 22, 2020, which claims the benefit of U.S.Provisional Application No. 62/796,065, filed Jan. 23, 2019, each ofwhich are incorporated by reference in their entirety.

BACKGROUND

When performing surgeries, surgeons often rely on pre-operativethree-dimensional images of the patient's anatomy such as computedtomography (CT) scan images. However, the usefulness of suchpre-operative images is limited because the images cannot be easilyintegrated into the operative procedure. For example, because the imagesare captured in a pre-operative session, the relative anatomicalpositions captured in the pre-operative images may vary from theiractual positions during the operative procedure. Furthermore, to makeuse of the pre-operative images during the surgery, the surgeon mustdivide attention between the surgical field and a display of thepre-operative images. Navigating between different layers of thepre-operative images may furthermore require significant attention thattakes away from the surgeon's focus on the operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example embodiment of an imaging system.

FIG. 2 is an example of a surgical environment employing the imagingsystem for mediated-reality assisted surgery.

FIG. 3A is a top plan view of an embodiment of a body patch withfiducial markers for aligning real-time operative images topre-operative scan images.

FIG. 3B is a cross-sectional view of an embodiment of a body patch withfiducial markers for aligning real-time operative images topre-operative scan images.

FIG. 4 is a flowchart illustrating an embodiment of a process forgenerating a mediated-reality view of a surgical site based onpre-operative images.

FIG. 5 illustrates an embodiment of a process for aligning pre-operativeimages captured prior to a patient being positioned for surgery topost-positioning images captured after a patient is positioned forsurgery.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

A mediated-reality system for surgical applications incorporatespre-operative images and real-time captured images of a surgical siteinto a visualization presented on a head-mounted display worn by asurgeon during a surgical procedure. The mediated-reality system tracksthe surgeon's head position and generates real-time images of thesurgical site from a virtual camera perspective corresponding to thesurgeon's head position to mimic the natural viewpoint of the surgeon.The mediated-reality system furthermore aligns the pre-operative imageswith the real-time images from the virtual camera perspective andpresents a mediated-reality visualization of the surgical site with thealigned pre-operative three-dimensional images or a selected portionthereof overlaid on the real-time images representing the virtual cameraperspective. The mediated-reality system thus enables the surgeon tovisualize the underlying three-dimensional anatomy of a patient prior tomaking an incision and throughout the procedure even when anatomicalfeatures may be occluded from the surgeon's view in the real-timeimages. The technology furthermore beneficially provides thevisualization in a manner that does not divert the surgeon's view fromthe surgical site, thus enhancing the surgeon's ability to perform theoperation with high efficiency and precision.

In an embodiment, a patch including fiducial markers is placed on thepatient's body and remains in places during the pre-operative imagingand after the patient is positioned for surgery. The fiducial markerscomprise a pattern that can be recognized by the mediated-reality systemin both the pre-operative images and in the real-time images capturedafter the patient is positioned for surgery. The mediated-reality systemaligns the pre-operative images with the real-time images based on thedetected positions of the fiducial markers visible in both sets ofimages. For example, the mediated-reality system may apply one or moretransformations to the pre-operative images that causes the detectedfiducial markers in the pre-operative images to align with thecorresponding fiducial markers detected in the real-time images.

In another embodiment, three-dimensional images such as ultrasound orfluoroscopic images may be captured after the patient is positioned forsurgery and may be utilized to predict changes in three-dimensionalpositions of anatomical features that occur between the pre-operativeimage scan and positioning the patient for surgery. The pre-operativeimages may then be warped to align the positioning of the anatomicalfeatures in the pre-operative image scan to the detected positions ofthe anatomical features seen in the three-dimensional images capturedafter the patient is positioned for surgery to compensate for changesthat may occur.

In yet further embodiments, a combination of fiducial markers andpost-positioning images may be used to align the pre-operative images tothe real-time images in the mediated-reality visualization. For example,the pre-operative and post-positioning images may be compared todetermine how the positions of the fiducial markers change inthree-dimensional space between the images, and a transformation may bederived that transforms the pre-operative images to align the fiducialmarkers with their respective locations in the post-positioning images.

In a specific example embodiment, a method generates a mediated realityview of a surgical site. Pre-operative images are received thatrepresent three-dimensional anatomy of a patient in a first position.Based on the pre-operative images, coordinates are identified in athree-dimensional pre-operative image space corresponding to locationsof fiducial markers present on a patch applied to the patient. Real-timeimages from a camera array are received after the patient is positionedfor surgery in a second position. Based on the real-time images,coordinates in a three-dimensional real-time image space are identifiedthat correspond to locations of the fiducial markers present on thepatch applied to the patient. A transformation is applied to thepre-operative images to substantially align the locations of thefiducial markers in the pre-operative images to the locations of thefiducial markers in the real-time images. The transformed pre-operativeimages are overlaid on the real-time images to generate the mediatedreality view, and the mediated reality view is provided to a displaydevice for display.

In another specific embodiment, a method generates a mediated realityview of a surgical site without necessarily relying on fiducial markerson a patch. In this embodiment, pre-operative images are received thatrepresent three-dimensional anatomy of a patient in a first position.Post-positioning images captured after the patient is positioned forsurgery in a second position are received. Based on the pre-operativeimages and the post-positioning images, a set of corresponding featuresvisible in the pre-operative images and the post-positioning images areidentified. A first transformation is applied to the pre-operativeimages to substantially align locations of the corresponding features inthe pre-operative images to respective locations in the post-positioningimages to generate initial transformed pre-operative images. The cameraarray captures real-time images of the patient and overlays the initialtransformed pre-operative images on the real-time images to generate aninitial mediated reality view which is provided to a display device fordisplay. Further details regarding the above-described embodiments anddescriptions of additional embodiments are provided below.

FIG. 1 illustrates an example embodiment of a mediated-reality system100 comprising an image processing device 110, a camera array 120, adisplay device 140, and an input controller 150. In alternativeembodiments, the mediated-reality system 100 may comprise additional ordifferent components.

The camera array 120 comprises a plurality of cameras 122 (e.g., acamera 122-1, a camera 122-2, . . . , a camera 122-N) that each capturerespective real-time images 190 of a scene 130. The cameras 122 may bephysically arranged in a particular configuration such that theirphysical locations and orientations are fixed relative to each other.For example, the cameras 122 may be structurally secured by a mountingstructure to mount the cameras 122 at predefined fixed locations andorientations. The cameras 122 of the camera array 120 may be positionedsuch that neighboring cameras may share overlapping views of the scene130. The cameras 122 in the camera array 120 may furthermore besynchronized to capture images 190 of the scene 130 substantiallysimultaneously (e.g., within a threshold temporal error). The cameraarray 120 may furthermore comprise one or more projectors 124 thatprojects a structured light pattern onto the scene 130. In anembodiment, the cameras 122 may comprise light-field cameras thatcapture light field information of the scene 130. Here, the cameras 122capture both the intensity of light and the directions of light raysrepresenting the scene. The light-field images 190 thus encode depthinformation and enable recreation of the scene as a three-dimensionalimage.

The image processing device 110 receives images 190 captured by thecamera array 120 and processes the images to synthesize an output imagecorresponding to a virtual camera perspective. Here, the output imagecorresponds to an approximation of an image of the scene 130 that wouldbe captured by a camera placed at an arbitrary position and orientationcorresponding to the virtual camera perspective. The image processingdevice 110 synthesizes the output image from a subset (e.g., one ormore) of the cameras 122 in the camera array 120, but does notnecessarily utilize images 190 from all of the cameras 122. For example,for a given virtual camera perspective, the image processing device 110may select a stereoscopic pair of images 190 from two cameras 122 thatare positioned and oriented to most closely match the virtual cameraperspective. The image processing device 110 may furthermore detect thestructured light projected onto the scene 130 by the projector toestimate depth information of the scene. The depth information may becombined with the images 190 from the cameras 122 to synthesize theoutput image as a three-dimensional rendering of the scene 130 as viewedfrom the virtual camera perspective. Alternatively, the structured lightprojector 124 may be omitted and the image processing device 110 mayderive the three-dimensional rendering solely from the images 190captured by the one or more cameras 122.

The virtual camera perspective may be controlled by an input controller150 that provides a control input corresponding to the location andorientation of the virtual camera perspective. The output imagecorresponding to the virtual camera perspective is outputted to thedisplay device 140 and displayed by the display device 140. The outputimage may be updated at a high frame rate to synthesize a videorepresentative of the virtual camera perspective. The image processingdevice 110 may furthermore beneficially process received inputs from theinput controller 150 and process the captured images 190 from the cameraarray 120 to generate output images corresponding to the virtualperspective in substantially real-time as perceived by a viewer of thedisplay device 140 (e.g., at least as fast as the frame rate of thecamera array 120).

The image processing device 110 may furthermore receive pre-operativeimages 170 representing a three-dimensional volume such as, for example,CT scan images, ultrasound images, or fluoroscopic images. As will bedescribed in further detail below, the image processing device 110 maydetect visual features in the pre-operative images 170 that correspondto visual features in the real-time images 190 captured by the cameraarray 120. The image processing device 110 may then apply one or moretransformations to the pre-operative images 170 to align the detectedfeatures in the pre-operative images 170 (or portion thereof) tocorresponding features detected in the real-time images 190. The imageprocessing device 110 may apply the one or more transformations on aframe-by-frame basis such that the pre-operative images 170 are alignedwith the real-time images 190 in each frame as the virtual perspectivechanges. The image processing device 110 overlays the pre-operativeimages 170 with the real-time images 190 to present a mediate-realityview that enables a surgeon to simultaneously visualize the surgicalsite and the underlying three-dimensional anatomy of a patientundergoing an operation.

In an embodiment, the scene 130 (e.g., a body of a surgical patient) maybe prepared with a patch 160 comprising fiducial markers prior tocapturing the pre-operative images 170. The image processing device 110may identify specific features of the fiducial markers that enable it toidentify correspondence between the features in the pre-operative images170 and the real-time images 190. The image processing device 110 mayapply the transformations to the pre-operative images 170 in a mannersuch that a pattern of the fiducial markers in the pre-operative images170 becomes aligned with the corresponding pattern visible in thereal-time images 190 from the camera array 120. For example, thepre-operative images 170 may be translated, rotated, and/or warped toalign the fiducial markers with corresponding fiducial markers in thereal-time images 190.

In an embodiment, the image processing device 110 optionally alsoreceives one or more post-positioning three-dimensional images 180captured of the scene 130 after the patient is positioned for surgery.The post-positioning images 180 may comprise, for example, ultrasound orfluoroscopic images captured once the patient is positioned for surgery.The image processing device 110 may utilize the post-positioning images180 in determining the transformation to apply to the pre-operativeimages 170 to align the pre-operative images 170 to the real-time images190. In an embodiment, the image processing device 110 may identify thefiducial markers or anatomical features in the post-positioning images180 and apply one or more transformations to the pre-operative images170 to align the pre-operative images 170 with the post-positioningimages 180. This transformation step may beneficially correct thepre-operative images 170 for a shift in the positioning of anatomicalelements to may have occurred in between capturing the pre-operativeimages 170 and positioning the patient for surgery.

The image processing device 110 may comprise a processor and anon-transitory computer-readable storage medium that stores instructionsthat when executed by the processor, carry out the functions attributedto the image processing device 110 as described herein.

The display device 140 may comprise, for example, a head-mounted displaydevice or other display device for displaying the output images receivedfrom the image processing device 110. In an embodiment, the inputcontroller 150 and the display device 140 are integrated into ahead-mounted display device and the input controller 150 comprises amotion sensor that detects position and orientation of the head-mounteddisplay device. The virtual perspective can then be derived tocorrespond to the position and orientation of the head-mounted displaydevice such that the virtual perspective corresponds to a perspectivethat would be seen by a viewer wearing the head-mounted display device.Thus, in this embodiment, the head-mounted display device can provide areal-time rendering of the scene as it would be seen by an observerwithout the head-mounted display. Alternatively, the input controller150 may comprise a user-controlled control device (e.g., a mouse,pointing device, handheld controller, gesture recognition controller,etc.) that enables a viewer to manually control the virtual perspectivedisplayed by the display device.

FIG. 2 illustrates an example embodiment of the mediated-reality system100 for a surgical application. Here, an embodiment of the camera array120 is positioned over the scene 130 (in this case, a surgical site) andcan be positioned via a swing arm 202 attached to a workstation 204. Theswing arm 202 may be manually moved or may be robotically controlled inresponse to the input controller 150. The workstation 204 may include acomputer to control various functions of the camera array 120 and thedisplay device 140, and may furthermore include a secondary display thatcan display a user interface for performing various configurationfunctions, or may mirror the display on the display device 140. Theimage processing device 120 and the input controller 150 may each beintegrated in the workstation 204, the display device 140, or acombination thereof.

FIG. 3A illustrates an example embodiment of a patch 160 applied to apatient 300 that may be utilized to align pre-operative images 170 withreal-time captured images 190 in a mediated-reality system 100. In anembodiment, the patch 160 comprises a thin flexible adhesive patch thatcan be applied to a patient's skin and conform to the contours of thepatient's body. The patch 160 may be positioned proximate to ananatomical structure of interest for the relevant surgical procedure(e.g., vertebrates 310). The patch 160 may include a pattern ofthree-dimensional fiducial markers 320 that can be uniquely identifiedby the image processing device 110 in both the pre-operative images 170and the real-time images 190 even when viewed from differentperspectives and under different lighting conditions. For example, thefiducial markers 320 may each comprise one or more raised surfacesforming a recognizable three-dimensional geometry. In an embodiment,each marker may be unique within a particular region of the patch 160.In an embodiment, a patch 160 comprises a grid (which may be uniform ornon-uniform) of fiducial markers each having a unique three-dimensionalgeometry and which can each be uniquely recognized in the pre-operativeimages 170 and the real-time images 190 and which can each be correlatedto a particular position on the human body. The fiducial markers 320 mayeach comprise materials of different densities that can be distinguishedby ultrasound or radio opaque images.

In an embodiment, the patch 160 may be partitioned into sectionsseparated by perforated boundaries. The perforations enable one or moresections of the patch 160 to be easily removed from the patient withoutremoving the entire patch 160. For example, in one use case, the surgeonmay remove a section of the patch over the desired incision locationafter the patient is positioned for surgery and the image processingdevice 110 performs an initial alignment computation. The remainingsections of the patch 160 that are not directly over the incisionlocation may remain in place. The image processing system 110 maycontinue to detect the fiducial markers 320 on the remaining portion ofthe patch 160 throughout the operation to update the alignment.

FIG. 3B illustrates example cross-sectional views of example fiducialmarkers 320 that may be integrated with a patch 160. As illustrated, thefiducial markers 320 have varying three-dimensional structures and mayinclude varying densities that enables the markers 320 to bedistinguished from each other in the real-time images 190, pre-operativeimages 170, and post-positioning images 180.

FIG. 4 illustrates an example embodiment of a process for generating amediated-reality visualization of a surgical site with pre-operativeimages 170 aligned and overlaid on real-time images 190. The imageprocessing device 110 receives 402 the pre-operative images 170. Theimage processing device 110 identifies 404 respective three-dimensionalcoordinates corresponding to the locations of the fiducial markers inthe pre-operative images 170. For example, the image processing device110 detects the fiducial markers in the pre-operative images 170 andmaps the fiducial markers to a first set of three-dimensionalcoordinates representing a location of the human body in thepre-operative image space. For example, the image processing device 110may obtain a predefined mapping between uniquely recognizable structuresof the fiducial markers and their corresponding locations on the patch160. The image processing device 110 receives 406 real-time images 190of a scene captured by the camera array 120. The image processing device110 identifies 408 respective three-dimensional coordinatescorresponding to locations of the fiducial markers in the real-timeimages 190. For example, the image processing device 110 detects thefiducial markers in the real-time images 190 and maps each fiducialmarker to a second set of three-dimensional coordinates representing alocation on the human body in the real-time image space. The imageprocessing device 110 then applies 410 one or more transformations tothe pre-operative images 170 that substantially aligns the pattern offiducial markers in the pre-operative images 170 to the pattern in thereal-time images 190. For example, in one embodiment, the imageprocessing device 110 performs an optimization algorithm to identify atransformation that minimizes a distance between transformed coordinatesof the pattern in the pre-operative images 170 with correspondingcoordinates of the pattern in the real-time images 190. The imageprocessing device 110 overlays 412 the transformed pre-operative imageson the real-time images to generate the mediated-reality view. Themediated-reality view is then provided 414 to the display device 140 fordisplay.

FIG. 5 illustrates an embodiment of a process for aligning pre-operativeimages 170 captured prior to a patient being positioned for surgery topost-positioning images 180 captured after a patient is positioned forsurgery. An image processing device 110 receives 502 the pre-operativeimages 170. The pre-operative images may comprise, for example, CT scanimages. The image processing device 110 receives 504 thepost-positioning images 180 captured after a patient is positioned forsurgery. The post-positioning images 180 may comprise, for example,ultrasound or fluoroscopic images. The image processing device 110identifies 506 a set of corresponding features (e.g., anatomicalfeatures or fiducial markers on a patch) visible in both thepre-operative images 170 and the post-positioning images 180. The imageprocessing device 110 then applies 508 a transformation to thepre-operative images 170 that substantially aligns the locations of thecorresponding features in the pre-operative images to the locations inthe post-positioning images 180. For example, the image processingdevice 110 may apply a translation, rotation, warping, or combinationthereof that minimizes a distance between transformed coordinates of thefeatures in the pre-operative images 170 with coordinates ofcorresponding features in the post-positioning images 180. Thetransformation beneficially compensates for shifts in the position ofanatomical features that may occur between capturing the pre-operativeimages 170 and positioning the patient for surgery. The transformedpre-operative images 170 may then be overlaid 510 on real-time images190 captured by the camera array 120 and provided 512 to a displaydevice 140 for display as a mediated-reality view of the surgical site.

In yet further embodiments, a transformation may be applied topre-operative images 170 based on both post-positioning images 180 andfiducial markers detected in the real-time images 190. For example, inone embodiment, a first transformation is applied to pre-operativeimages 170 in accordance with the process of FIG. 5 to obtain an initialtransformation. A second transformation is then applied according to theprocess of FIG. 4 to refine the alignment in each image frame based onthe pattern of fiducial markers detected in the pre-operative images 170and the real-time images 190.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for thedisclosed embodiments as disclosed from the principles herein. Thus,while particular embodiments and applications have been illustrated anddescribed, it is to be understood that the disclosed embodiments are notlimited to the precise construction and components disclosed herein.Various modifications, changes and variations, which will be apparent tothose skilled in the art, may be made in the arrangement, operation anddetails of the method and system disclosed herein without departing fromthe scope of the described embodiments.

1. A method for generating a mediated reality view of a surgical sitecomprising: receiving pre-operative images representingthree-dimensional anatomy of a patient in a first position; receivingpost-positioning images captured after the patient is positioned forsurgery in a second position; identifying, based on the pre-operativeimages and the post-positioning images, a set of corresponding featuresvisible in the pre-operative images and the post-positioning images;applying a first transformation to the pre-operative images tosubstantially align locations of the corresponding features in thepre-operative images to respective locations in the post-positioningimages to generate initial transformed pre-operative images; capturing,by a camera array, real-time images of the patient; overlaying theinitial transformed pre-operative images on the real-time images togenerate an initial mediated reality view; and providing the initialmediated reality view to a display device for display.
 2. The method ofclaim 1, wherein the features comprise anatomical features of a body ofthe patient.
 3. The method of claim 1, wherein the features comprisefiducial markers on a patch applied to a body of the patient.
 4. Themethod of claim 3, wherein the patch comprises a flexible adhesive patchpositioned proximate to an anatomical structure of interest.
 5. Themethod of claim 3, wherein the fiducial markers comprisethree-dimensional structures including one or more raised surfaces, andwherein each of the fiducial markers has a unique geometry relative toother fiducial markers on the patch.
 6. The method of claim 3, whereinthe fiducial markers are arranged in a grid on the patch.
 7. The methodof claim 3, wherein the fiducial markers comprise materials of differentdensities that distinguishable in the real-time images.
 8. The method ofclaim 3, wherein the patch comprises perforated boundaries for tearingthe patch into sections.
 9. The method of claim 1, further comprising:identifying, based on the pre-operative images, coordinates in athree-dimensional pre-operative image space corresponding to locationsof fiducial markers present on a patch applied to the patient;identifying, based on the real-time images, coordinates in athree-dimensional real-time image space corresponding to locations ofthe fiducial markers present on the patch applied to the patient;applying a second transformation to refine an alignment of the locationsof the fiducial markers in the pre-operative images to the locations ofthe fiducial markers in the real-time images to generated refinedpre-operative images; overlaying the refined pre-operative images on thereal-time images to generate a refined mediated reality view; andproviding the refined mediated reality view to the display device fordisplay.
 10. A non-transitory computer-readable storage medium storinginstructions for generating a mediated reality view of a surgical site,the instructions when executed causing a processor to perform stepsincluding: receiving pre-operative images representing three-dimensionalanatomy of a patient in a first position; receiving post-positioningimages captured after the patient is positioned for surgery in a secondposition; identifying, based on the pre-operative images and thepost-positioning images, a set of corresponding features visible in thepre-operative images and the post-positioning images; applying a firsttransformation to the pre-operative images to substantially alignlocations of the corresponding features in the pre-operative images torespective locations in the post-positioning images to generate initialtransformed pre-operative images; capturing, by a camera array,real-time images of the patient; overlaying the initial transformedpre-operative images on the real-time images to generate an initialmediated reality view; and providing the initial mediated reality viewto a display device for display.
 11. The non-transitorycomputer-readable storage medium of claim 10, wherein the featurescomprise anatomical features of a body of the patient.
 12. Thenon-transitory computer-readable storage medium of claim 10, wherein thefeatures comprise fiducial markers on a patch applied to a body of thepatient.
 13. The non-transitory computer-readable storage medium ofclaim 12, wherein the patch comprises a flexible adhesive patchpositioned proximate to an anatomical structure of interest.
 14. Thenon-transitory computer-readable storage medium of claim 12, wherein thefiducial markers comprise three-dimensional structures including one ormore raised surfaces, and wherein each of the fiducial markers has aunique geometry relative to other fiducial markers on the patch.
 15. Thenon-transitory computer-readable storage medium of claim 12, wherein thefiducial markers are arranged in a grid on the patch.
 16. Thenon-transitory computer-readable storage medium of claim 12, wherein thefiducial markers comprise materials of different densities thatdistinguishable in the real-time images.
 17. The non-transitorycomputer-readable storage medium of claim 12, wherein the patchcomprises perforated boundaries for tearing the patch into sections. 18.The non-transitory computer-readable storage medium of claim 10, furthercomprising: identifying, based on the pre-operative images, coordinatesin a three-dimensional pre-operative image space corresponding tolocations of fiducial markers present on a patch applied to the patient;identifying, based on the real-time images, coordinates in athree-dimensional real-time image space corresponding to locations ofthe fiducial markers present on the patch applied to the patient;applying a second transformation to refine an alignment of the locationsof the fiducial markers in the pre-operative images to the locations ofthe fiducial markers in the real-time images to generated refinedpre-operative images; overlaying the refined pre-operative images on thereal-time images to generate a refined mediated reality view; andproviding the refined mediated reality view to the display device fordisplay.
 19. A mediated reality system comprising: a camera array; adisplay device; a processor; and a non-transitory computer-readablestorage medium storing instructions for generating a mediated realityview of a surgical site, the instructions when executed causing aprocessor to perform steps including: receiving pre-operative imagesrepresenting three-dimensional anatomy of a patient in a first position;receiving post-positioning images captured after the patient ispositioned for surgery in a second position; identifying, based on thepre-operative images and the post-positioning images, a set ofcorresponding features visible in the pre-operative images and thepost-positioning images; applying a first transformation to thepre-operative images to substantially align locations of thecorresponding features in the pre-operative images to respectivelocations in the post-positioning images to generate initial transformedpre-operative images; capturing, by the camera array, real-time imagesof the patient; overlaying the initial transformed pre-operative imageson the real-time images to generate an initial mediated reality view;and providing the initial mediated reality view to the display devicefor display.
 20. The mediated reality system of claim 19, wherein thedisplay device comprises a head-mounted display device.